Restoration filter generation device and method, image processing device and method, imaging device, and non-transitory computer-readable medium

ABSTRACT

A restoration filter generation device which generates a restoration filter for performing a restoration process on luminance system image data, the restoration process being based on a point-image distribution in an optical system, the luminance system image data being image data relevant to luminance and being generated based on image data for each color of multiple colors, the restoration filter generation device including an MTF acquisition device which acquires a modulation transfer function MTF for the optical system; and a restoration filter generation device which generates the restoration filter based on the modulation transfer function MTF, the restoration filter suppressing an MTF value of image data for each color of the multiple colors to 1.0 or less at least in a region of a particular spatial frequency or less, the image data for each color of the multiple colors corresponding to the luminance system image data after the restoration process.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Division of co-pending U.S. application Ser. No.14/837,779 filed Aug. 27, 2015, which is a Continuation of PCTInternational Application No. PCT/JP2013/080647 filed on Nov. 13, 2013,which claims priority under 35 U.S.C§119(a) to Japanese PatentApplication No. 2013-042185 filed on Mar. 4, 2013. Each of the aboveapplication(s) is hereby expressly incorporated by reference, in itsentirety, into the present application.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a restoration filter generation deviceand method for generating a restoration filter to be used in arestoration process, an image processing device and method forperforming the restoration process using the restoration filter, animaging device including the image processing device, and anon-transitory computer-readable medium for generating the restorationfilter.

Description of the Related Art

In an image obtained by the imaging of a subject with an imaging devicesuch as a digital camera, an image degradation deriving from variousaberrations in an optical system (an image-taking lens and the like)sometimes appears. The image degradation by the aberrations can beexpressed by a point-image distribution function (PSF: Point SpreadFunction) and the like. Therefore, by generating a restoration filterbased on the degradation characteristic of the optical system such asthe PSF and performing a point-image restoration process (restorationprocess) on the image data using the restoration filter, it is possibleto reduce the image degradation.

Such a point-image restoration process is classified roughly into afrequency restoration process and a phase restoration process. Thefrequency restoration process equates the modulation transfer function(MTF) characteristic of the image degraded by the optical system. In thephase restoration process, a picture is moved depending on the frequencysuch that the asymmetric PSF form is restored to a point as much aspossible.

On this occasion, the aberration in the optical system such as theimage-taking lens differs depending on the wavelength, and therefore, itis desirable to perform the point-image restoration process usingrestoration filters that are different among the images for therespective colors of RGB. However, the process for each color results ina heavy computation load, and therefore, it has been considered toperform the point-image restoration process only on the luminancecomponent, which has a great visual effect. In this case, since the PSFform for the luminance cannot be defined, it is necessary to considersome sort of criterion and find a middle ground so as not to generate anadverse effect on the restoration performance.

An image processing device described in Japanese Patent ApplicationLaid-Open No. 2010-140442 (hereinafter referred to as PTL 1) calculatesPSFs for the respective colors of RGB, synthesizes the PSFs for therespective colors with previously defined weighting coefficients for therespective colors to calculates a PSF for a luminance system image (alsoreferred to as a luminance component image), and generates a restorationfilter for the luminance system image based on the PSF. Then, the imageprocessing device performs a point-image restoration process on theluminance system image, using the restoration filter, and thereafter,generates an RGB image based on the luminance system image after therestoration process and a color-difference system image (also referredto as a color component image) that is not a target of the point-imagerestoration process. Thereby, it is possible to reduce the imagedegradation such as false colors.

SUMMARY OF THE INVENTION

The image processing device in PTL 1 can averagely equate the RGB image(make the MTF values uniform with 1.0; equalize). However, the imageprocessing device in PTL 1 performs the point-image restoration processonly on the luminance system image, and therefore, there is a fear thatan overcorrection occurs for a particular wavelength (color). Theovercorrection herein means that at least any MTF value of the RGB imagecorresponding to the luminance system image after the point-imagerestoration process becomes greater than 1.0 (see the (D) portion ofFIG. 10).

An object of the present invention is to provide a restoration filtergeneration device and method for generating a restoration filter thatcan prevent the overcorrection of the MTF value of the image for eachcolor when the restoration process is performed on the luminance systemimage, an image processing device and method for performing therestoration process using the restoration filter, an imaging deviceincluding the image processing device, and a non-transitorycomputer-readable medium for generating the restoration filter.

A restoration filter generation device which attains the object of thepresent invention is a restoration filter generation device to generatea restoration filter for performing a restoration process on luminancesystem image data, the restoration process being based on a point-imagedistribution in an optical system, the luminance system image data beingimage data relevant to luminance and being generated based on image datafor each color of multiple colors, the image data for each color of themultiple colors being obtained by an imaging device including theoptical system, the restoration filter generation device including: anMTF acquisition device which acquires a modulation transfer function MTFfor the optical system; and a restoration filter generation device whichgenerates the restoration filter based on the modulation transferfunction MTF acquired by the MTF acquisition device, the restorationfilter suppressing an MTF value of image data for each color of themultiple colors to 1.0 or less at least in a region of a particularspatial frequency or less, the image data for each color of the multiplecolors corresponding to the luminance system image data after therestoration process.

According to the present invention, the MTF value of the image data foreach color of the multiple colors corresponding to the luminance systemimage data after the restoration process is suppressed to 1.0 or less,at least in the region of the particular spatial frequency or less.

It is preferable that the MTF acquisition device acquire the modulationtransfer function MTF for each color of the multiple colors, and therestoration filter generation device select a maximum value of the MTFvalue from the modulation transfer functions MTF for the respectivecolors, for each spatial frequency, determine a frequency characteristicof a Wiener filter, based on the maximum value of the MTF value for eachspatial frequency, and generate the restoration filter that achieves thefrequency characteristic in a luminance system. Thereby, a certaindegree of effect of suppressing the MTF value of the image data for eachcolor of the multiple colors corresponding to the luminance system imagedata after the point-image restoration process to 1.0 or less isexpected.

It is preferable that the restoration filter generation device includean amplification factor acquisition device which acquires anamplification factor of the MTF value for each spatial frequency withrespect to image data for each color of the multiple colors, theamplification factor corresponding to an amplification factor of the MTFvalue for each spatial frequency with respect to the luminance systemimage data after the restoration process, the image data for each colorof the multiple colors being generated by an inverse conversion processof the luminance system image data, the MTF acquisition device acquirethe modulation transfer function MTF for each color of the multiplecolors, and the restoration filter generation device determine frequencycharacteristics of restoration filters for the respective colors, basedon the modulation transfer functions MTF for the respective colorsrespectively, select a minimum value of the amplification factors of therestoration filters for the respective colors, for each spatialfrequency, based on the frequency characteristics, and generate therestoration filter based on the minimum value of the amplificationfactor for each spatial frequency and the acquisition result by theamplification factor acquisition device. The restoration process can beperformed using a restoration filter designed on the basis of a colorhaving a low MTF value for each spatial frequency, and therefore, it ispossible to suppress the MTF value of the image data for each color ofthe multiple colors corresponding to the luminance system image dataafter the restoration process to 1.0 or less. Further, it is possible toprevent the amplification of the noise for another color due to thefrequency characteristic of a restoration filter designed on the basisof a color having a high MTF in the particular spatial frequency.

It is preferable that the image data for the multiple colors containimage data for each color of RGB, the restoration filter generationdevice include an amplification factor acquisition device which acquiresan amplification factor of the MTF value for each spatial frequency withrespect to image data for each color of the multiple colors, theamplification factor corresponding to an amplification factor of the MTFvalue for each spatial frequency with respect to the luminance systemimage data after the restoration process, the image data for each colorof the multiple colors being generated by an inverse conversion processof the luminance system image data, the MTF acquisition device acquirethe modulation transfer function MTF for each color of the multiplecolors, and the restoration filter generation device calculate a filtercoefficient of a Wiener filter based on the modulation transfer functionMTF for each color of RGB and the acquisition result by theamplification factor acquisition device, and generate the restorationfilter based on the filter coefficient, the Wiener filter meeting acondition that the MTF value of the image data for each color after therestoration process is 1.0 or less. Thereby, it is possible to suppressthe MTF value of the image data for each color of the multiple colorscorresponding to the luminance system image data after the restorationprocess to 1.0 or less. Further, it is possible to averagely perform thebest frequency restoration while tolerating the noise emphasis for aparticular color in some degree, and to ensure that an excessiveemphasis of the MTF value does not occur for any color.

It is preferable that the image data for the multiple colors containimage data for each color of RGB, the MTF acquisition device acquire themodulation transfer function MTF for the G color, and the restorationfilter generation device determine a frequency characteristic of aWiener filter based on the modulation transfer function MTF for the Gcolor acquired by the MTF acquisition device, and generate therestoration filter that achieves the frequency characteristic in aluminance system. A certain degree of effect of suppressing the MTFvalue of the image data for each color of the multiple colorscorresponding to the luminance system image data after the restorationprocess to 1.0 or less at least in the region of the particular spatialfrequency or less is expected. Further, it is possible to simplify thecomputation process required for the generation of the restorationfilter. Moreover, it is possible to perform the generation of therestoration filter, without acquiring the modulation transfer functionsMTF for all colors with respect to the optical system.

It is preferable that the particular spatial frequency be equal to orless than a half of a Nyquist frequency of an imaging element includedin the imaging device. In this case, in the optical system, themodulation transfer function MTF for the G color is higher than themodulation transfer functions MTF for the other colors, in the frequencyregion that is equal to or less than the half of the Nyquist frequencyof the imaging element.

It is preferable that the MTF acquisition device acquire the modulationtransfer function MTF for the optical system including a lens unit thatmodulates a phase and extends a depth of field. This allows for asuitable image restoration in the range of an extended depth of field(focal depth), even for the restoration process in an EDoF system.

Further, an image processing device according to an aspect of thepresent invention includes: an image data generation device whichgenerates luminance system image data based on image data for each colorof multiple colors, the luminance system image data being image datarelevant to luminance, the image data for each color of the multiplecolors being obtained by an imaging device including an optical system;a restoration filter storage device which stores the restoration filtergenerated by the restoration filter generation device according to theabove each aspect; and a restoration processing device which performs arestoration process on the luminance system image data generated by theimage data generation device, using the restoration filter stored in therestoration filter storage device.

Further, an imaging device according to the present invention includes:an imaging device which outputs image data for each color of multiplecolors, the imaging device including an optical system; and the imageprocessing device according to the above aspect.

Further, a restoration filter generation method according to the presentinvention is a restoration filter generation method for generating arestoration filter for performing a restoration process on luminancesystem image data, the restoration process being based on a point-imagedistribution in an optical system, the luminance system image data beingimage data relevant to luminance and being generated based on image datafor each color of multiple colors, the image data for each color of themultiple colors being obtained by an imaging device including theoptical system, the restoration filter generation method including: anMTF acquisition step of acquiring a modulation transfer function MTF forthe optical system; and a restoration filter generation step ofgenerating the restoration filter based on the modulation transferfunction MTF acquired in the MTF acquisition step, the restorationfilter suppressing an MTF value of image data for each color of themultiple colors to 1.0 or less at least in a region of a particularspatial frequency or less, the image data for each color of the multiplecolors corresponding to the luminance system image data after therestoration process.

Further, an image processing method according to an aspect of thepresent invention includes: an image data generation step of generatingluminance system image data based on image data for each color ofmultiple colors, the luminance system image data being image datarelevant to luminance, the image data for each color of the multiplecolors being obtained by an imaging device including an optical system;and a restoration processing step of performing a restoration process onthe luminance system image data generated in the image data generationstep, using the restoration filter generated by the restoration filtergeneration method according to the above aspect.

Further, a non-transitory computer-readable medium according to thepresent invention is a non-transitory computer-readable medium recordinga program for generating a restoration filter for performing arestoration process on luminance system image data, the restorationprocess being based on a point-image distribution in an optical system,the luminance system image data being image data relevant to luminanceand being generated based on image data for each color of multiplecolors, the image data for each color of the multiple colors beingobtained by an imaging device including the optical system, the programcausing a computer to execute: an MTF acquisition step of acquiring amodulation transfer function MTF for the optical system; and arestoration filter generation step of generating the restoration filterbased on the modulation transfer function MTF acquired in the MTFacquisition step, the restoration filter suppressing an MTF value ofimage data for each color of the multiple colors to 1.0 or less at leastin a region of a particular spatial frequency or less, the image datafor each color of the multiple colors corresponding to the luminancesystem image data after the restoration process.

The restoration filter generation device and method, the imageprocessing device and method, the imaging device, and the transitorycomputer-readable medium according to the present invention perform thegeneration of the restoration filter that suppresses the MTF value ofthe image data for each color of the multiple colors corresponding tothe luminance system image data after the restoration process to 1.0 orless at least in the region of the particular spatial frequency or less,or the restoration process using the restoration filter, and therefore,it is possible to prevent the overcorrection of the MTF value of theimage for each color, even in the case of performing the restorationprocess on the luminance system image data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a restoration filter generation device and adigital camera that acquires a restoration filter from the restorationfilter generation device and performs a point-image restoration process.

FIG. 2 is a back perspective view of the digital camera.

FIG. 3 is a block diagram showing the electric configuration of thedigital camera and the restoration filter generation device.

FIG. 4 is a functional block diagram of an image processing circuit ofthe digital camera.

FIG. 5 is an explanatory diagram for explaining the point-imagerestoration process.

FIG. 6 is an explanatory diagram for explaining a restoration filtergeneration process.

FIG. 7 is a flowchart showing the flow of the generation process of arestoration filter.

FIG. 8 is a flowchart showing the flow of an image-taking process of thedigital camera.

FIG. 9 is an explanatory diagram for explaining a point-imagerestoration process.

FIG. 10 is an explanatory diagram for explaining a comparative examplein which a point-image restoration process is performed using aconventional restoration filter.

FIG. 11 is a block diagram showing the electric configuration of arestoration filter generation device according to a second embodiment.

FIG. 12 is an explanatory diagram for explaining an amplification factoracquisition unit.

FIG. 13 is an explanatory diagram for explaining a restoration filtergeneration process according to the second embodiment.

FIG. 14 is a flowchart showing the flow of the generation process of arestoration filter according to the second embodiment.

FIG. 15 is an explanatory diagram for explaining a function effect ofthe second embodiment.

FIG. 16 is an explanatory diagram for explaining a restoration filteraccording to a third embodiment.

FIG. 17 is a block diagram showing the electric configuration of arestoration filter generation device according to the third embodiment.

FIG. 18 is a flowchart showing the flow of the generation process of therestoration filter according to the third embodiment.

FIG. 19 is an explanatory diagram for explaining a function effect ofthe third embodiment.

FIG. 20 is a block diagram showing the electric configuration of arestoration filter generation device according to a fourth embodiment.

FIG. 21 is an explanatory diagram for explaining a restoration filtergeneration process according to the fourth embodiment.

FIG. 22 is a flowchart showing the flow of the generation process of arestoration filter according to the fourth embodiment.

FIG. 23 is a block diagram showing the electric configuration of animaging module including an EDoF optical system.

FIG. 24 is a schematic diagram of the EDoF optical system.

FIG. 25 is a flowchart showing the flow of a restoration process by arestoration processing unit that constitutes an imaging module.

FIG. 26 is a perspective view of a smart phone.

FIG. 27 is a block diagram showing the electric configuration of thesmart phone.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment<Configuration of Digital Camera>

In FIG. 1 and FIG. 2, a digital camera 2 corresponds to an imagingdevice according to the present invention. On the front surface of acamera body 3 of the digital camera 2, a lens barrel 4 including anoptical system and the like, a strobe light emission unit 5 and the likeare provided. On the top surface of the camera body 3, a shutter button6, a power switch 7 and the like are provided.

On the back surface of the camera body 3, a display unit 8, an operationunit 9 and the like are provided. In an image-taking standby state, thedisplay unit 8 functions as an electronic viewfinder, and displays alive view image (also referred to as a through image). Further, duringimage review, an image is reproduced and displayed on the display unit8, based on the image data recorded in a memory card 10.

The operation unit 9 is composed of a mode switch, a cross key, anexecution key and the like. The mode switch is operated for changing theoperation mode of the digital camera 2. The digital camera 2 has animage-taking mode for imaging a subject and obtaining taken image data,a review mode for review and display based on the taken image data, andthe like.

The cross key and the execution key are operated for the display ofvarious menu screens and setting screens on the display unit 8, themovement of a cursor displayed in these menu screens and settingscreens, the determination of a variety of setting of the digital camera2, and the like.

On the bottom surface of the camera body 3, although not shown, a cardslot into which a memory card 10 is loaded, and a mount cover that opensand closes the opening of the card slot are provided. In the memory card10, the taken image data obtained by the imaging of a subject isrecorded as an image file in various file formats.

The digital camera 2 having the above configuration performs apoint-image restoration process using a restoration filter 12 acquiredfrom a restoration filter generation device 11, for reducing the imagedegradation deriving from various aberrations in an optical system.Here, the acquisition of the restoration filter 12 by the digital camera2 is handled by a camera manufacturer.

As shown in FIG. 3, a camera CPU (Central Processing Unit) 13 of thedigital camera 2 sequentially executes a variety of programs and dataread from a memory 14, based on a control signal from the operation unit9, and integrally controls each unit of the digital camera 2.

In a ROM (Read Only Memory) region of the memory 14, which correspondsto a restoration filter storage device according to the presentinvention, the restoration filter 12 acquired from the restorationfilter generation device 11 is stored in addition to the above-describedvariety of programs. Further, a RAM (Random Access Memory) region of thememory 14 functions as a work memory for the process execution by thecamera CPU 13 and as a temporary storage location of a variety of data.

In the lens barrel 4, an optical system 17 composed of a zoom lens 15, afocus lens 16 and the like is embedded. The zoom lens 15 and the focuslens 16 are driven by a zoom mechanism 19 and a focus mechanism 20,respectively, and are moved back and forth along an optical axis OA ofthe optical system 17.

A mechanical shutter 18 includes a movable part (not shown) that movesbetween a closed position for blocking subject light incidence to acolor imaging element 23 and an open position for allowing the subjectlight incidence. The mechanical shutter 18 moves the movable part to therespective positions, and thereby, opens and shuts an optical path fromthe optical system 17 to the color imaging element 23. Further, themechanical shutter 18 includes a diaphragm for controlling the lightquantity of the subject light that enters the color imaging element 23.The mechanical shutter 18, the zoom mechanism 19 and the focus mechanism20 are operated and controlled by the camera CPU 13 through a lensdriver 24.

In the back of the optical system 17, the color imaging element 23 thatis a single-plate type is arranged. On the imaging surface of the colorimaging element 23, multiple pixels arranged in a matrix are formed in apredetermined pattern array (the Bayer array, the G stripe R/B fullchecker pattern, the X-Trans Array®, the honeycomb array, or the like).Each pixel includes a micro-lens, a color filter (in the example, an R(red), G (green) or B (blue) color filter), and a photodiode. The colorimaging element 23, which constitutes an imaging device according to thepresent invention together with the optical system 17, converts asubject image formed on the imaging surface by the optical system 17,into an electric output signal, and outputs it. Here, as the colorimaging element 23, each kind of imaging element of a CCD (ChargeCoupled Device) color imaging element, a CMOS (Complementary Metal OxideSemiconductor) color imaging element and the like is used. An imagingelement driver 25 controls the drive of the color imaging element 23,under the control by the camera CPU 13.

A signal adjustment circuit 28 performs various signal adjustmentprocesses on the output signal output from the color imaging element 23,and generates mosaic image data R1, G1, B1 (see FIG. 4) of RGBassociated with the color filter array of the color imaging element 23.Here, the signal adjustment circuit 28 is composed of, for example, aCDS/AGC (Correlated Double Sampling/Automatic Gain Control) circuit, anA/D conversion circuit and the like in the case where the color imagingelement 23 is a CCD type, and is composed of, for example, an amplifierand the like in the case of a CMOS type.

<Configuration of Image Processing Circuit>

An image processing circuit 29 corresponds to an image processing deviceaccording to the present invention. The image processing circuit 29performs a black level adjustment process, a white balance correctionprocess, a gamma correction process, a demosaic process, a YC conversionprocess, a point-image restoration process and the like, to the mosaicimage data R1, G1, B1 input from the signal adjustment circuit 28, andgenerates luminance system image data Y and color-difference systemimage data Cb, Cr (see FIG. 4). The luminance system image data Y andthe color-difference system image data Cb, Cr are temporarily stored ina VRAM (Video Random Access Memory) region of the memory 14 (the VRAMmay be provided separately).

The VRAM region has a memory area for the live view image in whichimages equivalent to two continuous fields are stored. The luminancesystem image data Y and color-difference system image data Cb, Cr storedin the VRAM region are sequentially output to the display unit 8.Thereby, the live view image is displayed on the display unit 8.

When the shutter button 6 is pushed down in the image-taking mode, acompression/decompression processing circuit 31 performs a compressionprocess on the luminance system image data Y and color-difference systemimage data Cb, Cr stored in the VRAM region. Further, thecompression/decompression processing circuit 31 performs a decompressionprocess on the compressed image data obtained from the memory card 10through a medium I/F 32. The medium I/F 32 performs the recording andreading of the compressed image data to and from the memory card 10 andthe like.

As show in FIG. 4, the image processing circuit 29 includes mainly aninput unit 35, a demosaic processing unit 36, a conversion unit (imagedata generation device) 37 and a point-image restoration processing unit(restoration processing device) 38. Here, although the image processingcircuit 29 includes also correction processing units to perform thewhite balance correction process, the gamma correction process and thelike, the illustration and explanation of these correction processingunits are omitted for preventing the complication of the explanation.

The input unit 35 outputs, to the demosaic processing unit 36, themosaic image data R1, G1, B1 for the respective colors of RGB input fromthe signal adjustment circuit 28. That is, the input unit 35 functionsas an input I/F to which the image data for the respective colorsobtained by the imaging with the color imaging element 23 is input.

The demosaic processing unit 36 performs a demosaic process (alsoreferred to as a synchronization process) for calculating the colorinformation for all of RGB on a pixel basis (for converting it into asynchronous system), based on the mosaic image data R1, G1, B1 for therespective colors, and generates RGB image data R2, G2, B2 composed ofthe color data of the three planes of RGB. The demosaic processing unit36 outputs the RGB image data R2, G2, B2 to the conversion unit 37.

The conversion unit 37 performs the YC conversion process on the RGBimage data R2, G2, B2, and generates the luminance system image data Yand the color-difference system image data Cb, Cr. The luminance systemimage data Y, for example, is generated in accordance with Formula[Y=0.3R+0.6G+0.1B]. In this formula, the contribution ratio of the Gcolor is 60%, and therefore, the G color is higher in contribution ratiothan the R color (a contribution ratio of 30%) and the B color (acontribution ratio of 10%). Therefore, the G color is the color of thethree primary colors that most contributes to the luminance signal.

Here, in the embodiment, the luminance system image data Y is explainedtaking as an example the value of the luminance signal in a color spaceto be expressed by “Y, Cb, Cr”, but is not particularly limited if it isthe data contributing to the luminance of an image, and means a varietyof data having the information relevant to the luminance of a capturedimage. The examples include the data indicating the brightness L in aCIELAB (Commission internationale de l'eclairage) color space, thehighest data in the contribution ratio for obtaining the luminancesignal, the data corresponding to the color filter with the color thatmost contributes to the luminance, and the like.

The point-image restoration processing unit 38 reads the restorationfilter 12 stored in the memory 14, and performs a point-imagerestoration process (a restoration process according to the presentinvention) on the luminance system image data Y using the restorationfilter 12. For decreasing the load of the computation process, thepoint-image restoration process is performed only on the luminancesystem image data Y, which has a great visual effect. By performing thepoint-image restoration process, the blur of the image is corrected asshown in FIG. 5.

As shown in the (A) portion of FIG. 5, a point-image (optical image)transmitted by the optical system 17 is formed on the imaging surface ofthe color imaging element 23, as a large point-image (a blurred image),but by the point-image restoration process, is restored to a smallpoint-image (an image giving a high resolution feeling) as shown in the(B) portion of FIG. 5.

Here, as described above, the point-image restoration process isclassified roughly into the frequency restoration process and the phaserestoration process. Since an object of the present application is toprevent the overcorrection of the MTF value of the image for each colorof RGB when the point-image restoration process is performed on theluminance system image data Y, only the frequency restoration process isexplained as the point-image restoration process, and the explanation ofthe phase restoration process is omitted.

<Configuration of Restoration Filter Generation Device>

Returning to FIG. 3, the restoration filter generation device 11generates the restoration filter 12 to be used in the point-imagerestoration process of the digital camera 2. The restoration filtergeneration device 11 includes a device CPU 40, an operation unit 41, astorage 43 and a communication I/F 44.

The device CPU 40 appropriately reads various programs from the storage43 to execute it, based on an operation instruction input to theoperation unit 41, and thereby, integrally controls the whole of thedevice. Further, the operation unit 41 is a keyboard or a mouse, forexample.

In the storage 43, a restoration filter generation program 46corresponding to a program according to the present invention, lens MTFs47R, 47G, 47B corresponding to the modulation transfer functions MTF forthe respective colors of RGB in the optical system 17, and the like arestored.

The communication I/F 44 is connected with an MTF measurement device 49that measures the lens MTFs of the optical system 17. Here, themeasurement method for the lens MTFs 47R, 47G, 47B by the MTFmeasurement device 49 is a known technology, and therefore, the specificexplanation is omitted. The communication I/F 44, under the control bythe device CPU 40, acquires the lens MTFs 47R, 47G, 47B corresponding tothe optical system 17, from the MTF measurement device 49, and storesthem in the storage 43.

Further, the communication I/F 44 can be connected with a communicationI/F (not shown) of the digital camera 2 through various communicationcables and communication lines (including wireless), and sends, to thedigital camera 2, the restoration filter 12 generated by the device CPU40. Thereby, the restoration filter 12 is stored in the memory 14 of thedigital camera 2.

<Generation Process of Restoration Filter>

When a restoration filter generation operation is performed in theoperation unit 41, the device CPU 40 reads a restoration filtergeneration program 46 from the storage 43 to execute it, and thereby,functions as an MTF acquisition unit 50 and a restoration filtergeneration unit (restoration filter generation device) 51.

The MTF acquisition unit 50 functions as an MTF acquisition deviceaccording to the present invention, together with the above-describedcommunication I/F 44, and outputs, to the restoration filter generationunit 51, the lens MTFs 47R, 47G, 47B for the respective colors of RGBstored in the storage 43.

Based on the lens MTFs 47R, 47G, 47B, the restoration filter generationunit 51 generates the restoration filter 12 that can suppress theovercorrection of the MTF value of the RGB image data (hereinafter,referred to as merely the “restoration RGB image data”) corresponding tothe luminance system image data Y after the point-image restorationprocess. Here, the restoration RGB image data corresponds to the RGBimage data that is inversely converted from the luminance system imagedata Y after the point-image restoration process and thecolor-difference system image data Cb, Cr that are not targets of thepoint-image restoration process. Further, the suppression of theovercorrection of the MTF value of the restoration RGB image data devicethat the MTF value of the restoration RGB image data is suppressed to1.0 or less. In the following, the generation of the restoration filter12 by the restoration filter generation unit 51 is specificallyexplained using FIG. 6. Here, the lens MTF and the restoration filterare expressed as a two-dimensional function with respect to the spatialfrequency, but in the specification, are expressed as a one-dimensionalfunction with respect to the spatial frequency, for preventing thecomplication of the drawings.

As shown in the (A) portion of FIG. 6, the restoration filter generationunit 51 compares the magnitudes of the MTF values of the lens MTFs 47R,47G, 47B for each spatial frequency, based on the frequencycharacteristics (spatial frequency-MTF value) of the lens MTFs 47R, 47G,47B. Here, the frequency characteristic of the lens MTF indicates therelation between the spatial frequency and the MTF value that changesdepending on the spatial frequency, and the MTF value decreases as thespatial frequency increases. Then, as shown in the (B) portion of FIG.6, the restoration filter generation unit 51 selects, for each spatialfrequency, the maximum value (shown by the solid line) of the MTF valuefrom the lens MTFs 47R, 47G, 47B for the respective colors, anddetermines a maximum lens MTF. Specifically, when the frequencycharacteristics of the lens MTFs 47R, 47G, 47B are represented by anMTF_(R) (ω_(x), ω_(y)), an MTF_(G) (ω_(x), ω_(y)) and an MTF_(B) (ω_(x),ω_(y)) respectively, the restoration filter generation unit 51determines an MTF (ω_(x), ω_(y)), which is the frequency characteristicof the maximum lens MTF, by the following Formula (1).

[Formula 1]

MTF(ω_(x),ω_(y)=max{MTF_(R)(ω_(x),ω_(y)),MTF_(G)(ω_(x),ω_(y)),MTF_(B)(ω_(x),ω_(y))}  (1)

Subsequently, as shown in the (C) portion of FIG. 6, the restorationfilter generation unit 51 computes the computation formula of a Wienerfilter to be used in the generation of the restoration filter, based onthe maximum lens MTF (MTF(ω_(x), ω_(y))). Thereby, the frequencycharacteristic (spatial frequency-amplification factor) of the Wienerfilter is determined. Here, the spatial frequency of the Wiener filterindicates the relation between the spatial frequency and theamplification factor corresponding to the decrease in the MTF value foreach spatial frequency. On this occasion, the frequency characteristicof the restoration filter corresponding to the maximum lens MTF isdetermined, because the Wiener filter is the restoration filtercorresponding to the maximum lens MTF. Here, the computation formulaitself of the Wiener filter is known (for example, see Formulas (8) and(9) described later), and the method for determining the frequencycharacteristic of the restoration filter corresponding to the lens MTFusing the computation formula is also known. Therefore, the specificexplanation is omitted herein.

As shown in the (D) portion of FIG. 6, the filter coefficient of therestoration filter 12 that achieves the frequency characteristic in theluminance system is determined based on the frequency characteristic ofthe Wiener filter corresponding to the maximum lens MTF. Here, as themethod for designing, from the frequency characteristic (spatialfrequency-amplification factor), the restoration filter (filtercoefficient) that achieves this, various known methods can be used.

Here, it is preferable that the restoration filter 12 in the luminancesystem is generated from the lens MTF with respect to the luminance.However, it is difficult to exactly determine the lens MTF with respectto the luminance. Therefore, in the embodiment, the restoration filter12 in the luminance system is generated using the maximum lens MTF shownin the (B) portion of FIG. 6, instead of the lens MTF with respect tothe luminance. The restoration filter 12 defines the correction amount(amplification factor) corresponding to the MTF value of the colorhaving the highest frequency response for each spatial frequency, andtherefore, the overcorrection is prevented at least for the RGB signalcomponent (Y=0.3R+0.6G+0.1B) of the luminance system image data Y afterthe point-image restoration process.

The restoration filter 12 generated by the restoration filter generationunit 51 is output to the digital camera 2 through the communication I/F44, and thereafter, is stored in the memory 14.

<Action of Restoration Filter Generation Device According to FirstEmbodiment>

Next, the actions of the digital camera 2 and restoration filtergeneration device 11 having the above configurations are explained.First, the generation process of the restoration filter 12 by therestoration filter generation device 11 is explained. Here, before thegeneration process of the restoration filter 12 is started, themeasurement of the lens MTFs 47R, 47G, 47B by the MTF measurement device49 and the storing of the respective lens MTFs 47R, 47G, 47B in thestorage 43 are executed.

When the restoration filter generation operation is performed in theoperation unit 41, the device CPU 40 reads the restoration filtergeneration program 46 in the storage 43 to execute it, and thereby,functions as the MTF acquisition unit 50 and the restoration filtergeneration unit 51.

As shown in FIG. 7, the MTF acquisition unit 50 acquires the lens MTFs47R, 47G, 47B from the storage 43, and outputs these lens MTFs 47R, 47G,47B to the restoration filter generation unit 51 (step S1, MTFacquisition step).

As shown in the (A) portion and (B) portion of FIG. 6 and Formula (1)described above, the restoration filter generation unit 51 selects, foreach spatial frequency, the maximum value of the MTF value from the lensMTFs 47R, 47G, 47B for the respective colors, and determines the MTF(ω_(x), ω_(y)) of the maximum lens MTF (step S2). Then, as shown in the(C) portion of FIG. 6, the restoration filter generation unit 51computes the computation formula of the Wiener filter based on the MTF(ω_(x), ω_(y)) of the maximum lens MTF, and determines the frequencycharacteristic of the Wiener filter corresponding to the maximum lensMTF (step S3).

Subsequently, the restoration filter generation unit 51 determines thefilter coefficient of the restoration filter 12 that achieves thefrequency characteristic of the Wiener filter in the luminance system.Thereby, in the restoration filter generation unit 51, the restorationfilter 12 is generated (step S4, restoration filter generation step).The restoration filter 12 is stored in the storage 43.

During the manufacture of the digital camera 2, the device CPU 40 readsthe restoration filter 12 stored in the storage 43, and outputs it tothe communication I/F 44. The communication I/F 44 outputs therestoration filter 12 to the digital camera 2, which is connectedthrough a communication cable not illustrated. Thereby, the restorationfilter 12 is stored in the memory 14 (step S5).

<Action of Digital Camera According to First Embodiment>

Next, the action of the digital camera 2 to perform the point-imagerestoration process using the restoration filter 12 is explained.

As shown in FIG. 8, when the operation mode of the digital camera 2 isset to the image-taking mode on the operation unit 9 (step S9), thecamera CPU 13 drives the color imaging element 23 through the imagingelement driver 25 and starts the imaging process (step S10). Themechanical shutter 18 is opened and shut at a predetermined shutterspeed, and signal charges are accumulated in each of the pixels of thecolor imaging element 23. Then, under the control by the imaging elementdriver 25, the signal is output from each pixel of the color imagingelement 23.

The signal adjustment circuit 28 generates the mosaic image data R1, G1,B1 of RGB by performing various signal adjustment processes on theoutput signal output from the color imaging element 23, and outputs therespective mosaic image data R1, G1, B1 to the image processing circuit29 (step S11). The respective mosaic image data R1, G1, B1 are input tothe demosaic processing unit 36 through the input unit 35. The demosaicprocessing unit 36 generates the RGB image data R2, G2, B2 by performingthe demosaic process on the mosaic image data R1, G1, B1, and outputsthe RGB image data R2, G2, B2 to the conversion unit 37 (step S12).

The conversion unit 37 generates the luminance system image data Y andthe color-difference system image data Cb, Cr by performing the YCconversion process on the RGB image data R2, G2, B2 (step S13, imagedata generation step). Then, the conversion unit 37 outputs theluminance system image data Y to the point-image restoration processingunit 38.

When the operation mode of the digital camera 2 is set to theimage-taking mode, the point-image restoration processing unit 38 readsthe restoration filter 12 that is previously stored in the memory 14.Then, as shown in the above-described (A) and (B) portion of FIG. 5, thepoint-image restoration processing unit 38 performs the point-imagerestoration process on the luminance system image data Y input from theconversion unit 37, using the restoration filter 12 (step S14,restoration processing step). The luminance system image data Y afterthe point-image restoration process, and the color-difference systemimage data Cb, Cr that are not targets of the point-image restorationprocess, are stored in the VRAM region of the memory 14, as the takenimage data.

The image processing circuit 29 (the camera CPU 13 may be adopted)generates the live view image data from the taken image data stored inthe VRAM region of the memory 14, and outputs it to the display unit 8.Thereby, the live view image is displayed on the display unit 8.Thereafter, the processes of step S10 to step S14 are repeatedlyexecuted until the shutter button 6 is pushed down (NO in step S15).

When the shutter button 6 is pushed down (YES in step S15), the takenimage data (the luminance system image data Y after the point-imagerestoration process, and the color-difference system image data Cb, Cr)equivalent to one frame is generated in the image processing circuit 29,and is stored in the VRAM region of the memory 14. The taken image datais compressed in the compression/decompression processing circuit 31,and thereafter, is recorded in the memory card 10 through the medium I/F32 (step S16). Thereafter, the above-described processes are repeatedlyexecuted until the image-taking mode is finished (step S17). Here, inthe embodiment, the point-image restoration process is performed whenthe live view image is displayed, but the point-image restorationprocess does not need to be performed when the live view image isdisplayed (step S14 is omitted). In this case, when the shutter button 6is pushed down, the processes of step S11 to step S14 may be executed tothe output signal output from the color imaging element 23, and thereby,the taken image data for the recording may be generated. Thereby, it ispossible to decrease the load of the computation process.

Function Effect of First Embodiment

On this occasion, as shown in the (A) portion and (B) portion of FIG. 9,in the embodiment, the point-image restoration process is performed,using the restoration filter 12 generated with use of the maximum lensMTF, that is, using the restoration filter 12 having the correctionamount (amplification factor) corresponding to the lens MTF value of thecolor having the highest frequency response for each spatial frequency.As described above, this results in the prevention of the overcorrectionof the MTF value of the RGB signal component (Y=0.3R+0.60+0.1B) of theluminance system image data Y after the point-image restoration process.

The prevention of the overcorrection of the MTF value of the RGB signalcomponent of the luminance system image data Y in this way makes acertain degree of effect of suppressing the overcorrection of the MTFvalue for each color of the restoration RGB image data corresponding tothe luminance system image data Y. Therefore, as shown in the (C)portion of FIG. 9, by performing the point-image restoration processusing the restoration filter 12 generated with use of the maximum lensMTF, a certain degree of effect of suppressing the MTF value for eachcolor of the restoration RGB image data corresponding to the luminancesystem image data Y after the point-image restoration process to 1.0 orless is expected.

In contrast, in a comparative example shown in the (A) portion to (D)portion of FIG. 10, the average value of the MTF values of the lens MTFs47R, 47G, 47B for the respective colors is determined for each spatialfrequency (the (A) portion and (B) portion of FIG. 10), and based on anaverage lens MTF that is the average value, the restoration filter inthe luminance system is generated (the (C) portion of FIG. 10). In thiscase, there is a fear that the overcorrection of the MTF value occursdepending on the color of the restoration RGB image data. For example,in the (B) portion of FIG. 10, the lens MTF 47B has a lower value thanthe average lens MTF, and therefore, there is no fear that theovercorrection of the MTF value of the restoration B image data occurs,even when the point-image restoration process is performed with therestoration filter shown in the (C) portion of FIG. 10. However, thelens MTFs 47G, 47R basically has a higher value than the average lensMTF. Therefore, when the point-image restoration process is performedwith the restoration filter shown in the (C) portion of FIG. 10, thereis a fear that the G signal and the R signal are excessively emphasizedas shown in the (D) portion of FIG. 10 and thereby the overcorrection ofthe MTF values of the restoration R image data and the restoration Gimage data occurs.

On the contrary to such a comparative example, in the embodiment, thepoint-image restoration process is performed using the restorationfilter 12 generated with use of the maximum lens MTF, and thereby, it ispossible to reduce the occurrence of overcorrection of the MTF value foreach color of the restoration RGB image data, relative to thecomparative example.

Second Embodiment

Next, a restoration filter generation device according to a secondembodiment of the present invention is explained. The above restorationfilter 12 according to the first embodiment shown in the (B) portion ofFIG. 9 is designed to correspond to the lens MTFs 47R, 47G also, whichare higher in the MTF value than the lens MTF 47B in a high frequencyregion. Therefore, since the restoration filter 12 is designed such thatthe amplification factor in a high frequency region (a low SN ratioregion in FIG. 15) is high, there is a fear that an excessive frequencyemphasis is applied in a region where the SN ratio of the B signal islow (hereinafter, referred to as a low SN ratio region, see FIG. 15) andthereby the noise of the B signal is amplified.

Hence, the restoration filter generation device according to the secondembodiment generates a restoration filter that can prevent the sideeffect that the frequency characteristic of a restoration filterdesigned on the basis of a color having a high MTF value amplifies thenoise for another color.

<Configuration of Restoration Filter Generation Device>

As shown in FIG. 11, a restoration filter generation device 55 accordingto the second embodiment basically has the same configuration as thefirst embodiment, except that a device CPU 56 is included, and that arestoration filter generation program 57 (corresponding to a programaccording to the present invention) and a function d_(K) (χ)corresponding to an amplification factor according to the presentinvention are stored in the storage 43. Therefore, for constituentshaving the same functions and configurations as the above firstembodiment, the same reference characters and the same referencenumerals are assigned, and the explanation is omitted.

When the restoration filter generation operation is performed in theoperation unit 41, the device CPU 56 reads the restoration filtergeneration program 57 from the storage 43 to execute it, and thereby,functions as an amplification factor acquisition unit (amplificationfactor acquisition device) 59 and a restoration filter generation unit(restoration filter generation device) 60, in addition to theabove-described MTF acquisition unit 50.

When an amplification factor χ (≧0) of the restoration filter in theluminance system is given, the amplification factor acquisition unit 59determines, by a computation process, the function d_(K) (χ), Kε{R, G,B} that gives the amplification factor of each MTF value of therestoration RGB image data, and stores it in the storage 43. Thecomputation process of determining the function d_(K) (χ) and thestoring in the storage 43 are previously executed before the generationof the restoration filter. In the following, the computation process ofthe function d_(K) (χ) is explained.

<Computation Process of Function d_(K) (χ)>

As shown in FIG. 12, a computation processing unit 59 a of theamplification factor acquisition unit 59 estimates, using a restorationmodel 61 in the luminance system, the gains (the levels of the influenceon the MTF) for the respective planes of RGB of the restoration RGBimage data when the restoration filter in the luminance systemmultiplies the luminance gain by “χ.” Here, for the restoration model61, a conversion unit 61 a, a formula 61 b and an inverse conversionunit 61 c are used, for example.

The conversion formula 61 a is an arbitrary conversion formula thatconverts input image data (signal, pixel value) r_(in), g_(in), b_(in)into the luminance system image data (signal) Y and the color-differencesystem image data (signal) Cb, Cr, and is represented by “M.” Theformula 61 b is utilized for the above-described estimation, and isdifferent from the actual restoration filter. The inverse conversionunit 61 c is an arbitrary conversion formula that inversely converts theluminance system image data Y and the color-difference system image dataCb, Cr into output image data (signal, pixel value) r_(out), g_(out),b_(out), and is represented by “M⁻¹.” The relation between the inputimage data r_(in), g_(in), b_(in), and the output image data r_(out),g_(out), b_(out) in the case of using the restoration model 61 isexpressed as the following Formula (2).

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 2} \rbrack & \; \\{\begin{pmatrix}r_{out} \\g_{out} \\b_{out}\end{pmatrix} = {{M^{- 1}\begin{pmatrix}x & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{pmatrix}}{M\begin{pmatrix}r_{in} \\g_{in} \\b_{in}\end{pmatrix}}}} & (2)\end{matrix}$

Here, generally, the conversion from an RGB space into a YCbCr spaceinvolves a gamma correction, and therefore, a non-linear conversion isnecessary. However, in some cases, a linear conversion is adopteddepending on the implementation form of the luminance system restorationprocess. By utilizing this, the MTF amplification factors for therespective planes of RGB are expressed as the following Formula (3).

[Formula 3]

d _(R)(x|r _(in) ,g _(in) ,b _(in))=r _(out) /r _(in)

d _(G)(x|r _(in) ,g _(in) ,b _(in))=g _(out) /g _(in)

d _(B)(x|r _(in) ,g _(in) ,b _(in))=b _(out) /b _(in)  (3)

As shown in the above Formula (3), the MTF amplification factors for therespective planes of RGB are influenced by not only the filteramplification facto “χ” but also the input image data r_(in), g_(in),b_(in). However, from the standpoint of computation amount reduction, itis difficult to apply a restoration filter for which all of the inputimage data r_(in), g_(in), b_(in) are exactly considered. Therefore, asshown in the following Formula (4), the computation processing unit 59 adetermines d_(K) (χ) by assuming a prior distribution w (r_(in), g_(in),b_(in)) of the input image data r_(in), g_(in), b_(in) and taking anexpected value with respect to this. Then, the computation processingunit 59 a stores the function d_(K) (χ) in the storage 43. Here, “Z” inFormula (4) is a normalization constant resulting from integrating theprior distribution w (r_(in), g_(in), b_(in)) over the whole domain.

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 4} \rbrack & \; \\{{d_{K}(ϰ)} = {\frac{1}{Z}{\int{\int{\int{{d_{K}( {{ϰr_{in}},g_{in},b_{in}} )}{w( {r_{in},g_{in},b_{in}} )}{dr}_{in}{dg}_{in}{db}_{in}}}}}}} & (4)\end{matrix}$

Returning to FIG. 11, in the generation of the restoration filter, theamplification factor acquisition unit 59 acquires the function d_(K) (χ)from the storage 43, and outputs it to the restoration filter generationunit 60.

<Restoration Filter Generation Process>

Similarly to the restoration filter 12 according to the firstembodiment, the restoration filter generation unit 60, based on the lensMTFs 47R, 47G, 47B, generates a restoration filter 62 in the luminancesystem that suppresses the overcorrection of the MTF value of therestoration RGB image data, that is, that suppresses the MTF value to1.0 or less.

As shown in the (A) portion and (B) portion of FIG. 13, based on thefrequency characteristics of the lens MTFs 47R, 47G, 47B, therestoration filter generation unit 60 calculates the frequencycharacteristics (spatial frequency-amplification factor) of therestoration filters for the respective colors of RGB corresponding tothe lens MTFs 47R, 47G, 47B respectively. The restoration filters forthe respective colors are a restoration filter 64R for the R color, arestoration filter 64G for the G color and a restoration filter 64B forthe B color.

Specifically, when the frequency characteristics of the restorationfilters 64R, 64G, 64B for the respective colors at a frequency (ω_(x),ω_(y)) are represented by h_(R) (ω_(x), ω_(y)), h_(G) (ω_(x), ω_(y)) andh_(B) (ω_(x), ω_(y)) respectively, the restoration filter generationunit 60 determines these by the following computation formula (5) of theWiener filter, respectively. Here, Kε{R, G, B} is held. Further, S_(K)(ω_(x), ω_(y)) and N_(K) (ω_(x), ω_(y)) represent the signal power andnoise power for each color, respectively, and already-known values thatare previously determined.

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 5} \rbrack & \; \\{{h_{K}( {\omega_{x},\omega_{y}} )} = \frac{{MTF}_{K}^{*}( {\omega_{x},\omega_{y}} )}{{{{MTF}_{K}( {\omega_{x},\omega_{y}} )}}^{2} + {{N_{K}( {\omega_{x},\omega_{y}} )}/{S_{K}( {\omega_{x},\omega_{y}} )}}}} & (5)\end{matrix}$

Subsequently, the restoration filter generation unit 60 selects, foreach spatial frequency, the minimum value of the amplification factorsof the restoration filters 64R, 64G, 64B for the respective colors, andgenerates the restoration filter 62 in the luminance system based on theminimum value of the amplification factor for each spatial frequency andthe function d_(K) (χ). Specifically, when the frequency characteristicof the restoration filter 62 in the luminance system is represented by f(ω_(x), ω_(y)), the restoration filter generation unit 60 calculates thefrequency characteristic f (ω_(x), ω_(y)) shown in the (C) portion ofFIG. 13, based on the following Formula (6), and determines the filtercoefficient of the restoration filter 62 that achieves the frequencycharacteristic f (ω_(x), ω_(y)). Thereby, the restoration filter 62 isgenerated. Similarly to the first embodiment, the restoration filter 62is stored in the memory 14 of the digital camera 2.

[Formula 6]

f(ω_(x),ω_(y))=min{d _(R) ⁻¹(h _(R)(ω_(x),ω_(y))),d _(G)⁻¹(ω_(x),ω_(y))),d _(B) ⁻¹(h _(B)(ω_(x),ω_(y)))}  (6)

<Action of Restoration Filter Generation Device According to SecondEmbodiment>

Next, the generation process of the restoration filter 62 by therestoration filter generation device 55 having the above configurationis explained using FIG. 14. Here, before the generation process of therestoration filter 62 is started, the measurement and storing of thelens MTFs 47R, 47G, 47B by the MTF measurement device 49 and thecalculation and storing of the function d_(K) (χ) by the computationprocessing unit 59 a are executed.

When the restoration filter generation operation is performed in theoperation unit 41, the device CPU 56 reads the restoration filtergeneration program 57 in the storage 43 to execute it, and thereby,functions as the MTF acquisition unit 50, the amplification factoracquisition unit 59 and the restoration filter generation unit 60.

The MTF acquisition unit 50 acquires the lens MTFs 47R, 47G, 47B fromthe storage 43, and outputs the respective lens MTFs 47R, 47G, 47B tothe restoration filter generation unit 60 (step S20). Further, theamplification factor acquisition unit 59 acquires the function d_(K) (χ)from the storage 43, and outputs the function d_(K) (χ) to therestoration filter generation unit 60 (step S20).

The restoration filter generation unit 60 calculates each of thefrequency characteristics h_(R) (ω_(x), ω_(y)), h_(G) (ω_(x), ω_(y)) andh_(B) (ω_(x), ω_(y)) of the restoration filters 64R, 64G, 64B for therespective colors by the above-described Formula (5), based on the lensMTFs 47R, 47G, 47B acquired from the MTF acquisition unit 50 and thealready-known S_(K) and N_(K) (step S21).

Subsequently, the restoration filter generation unit 60 calculates thefrequency characteristic f (ω_(x), ω_(y)) of the restoration filter 62in the luminance system by the above-described Formula (6), based on thefrequency characteristics h_(R) (ω_(x), ω_(y)), h_(G) (ω_(x), ω_(y)) andh_(B) (ω_(x), ω_(y)) and the function d_(K) (χ). That is, therestoration filter generation unit 60 calculates the frequencycharacteristic f (ω_(x), ω_(y)) of the restoration filter 62, based onthe minimum value of the amplification factors of the restorationfilters 64R, 64G, 64B for the respective colors at each spatialfrequency and the function d_(K) (χ) (step S22).

The restoration filter generation unit 60 determines the filtercoefficient of the restoration filter 62 that achieves the frequencycharacteristic f (ω_(x), ω_(y)), based on the calculation result of thefrequency characteristic f (ω_(x), ω_(y)). Thereby, in the restorationfilter generation unit 60, the restoration filter 62 is generated (stepS23). The restoration filter 62 is stored in the storage 43. Thereafter,similarly to the first embodiment, the restoration filter 62 is storedin the memory 14 (step S5).

<Action of Digital Camera According to Second Embodiment>

The imaging process in the digital camera 2 according to the secondembodiment is basically the same as the flow of the imaging process inthe first embodiment shown in FIG. 8, except that the point-imagerestoration process is performed using the restoration filter 62 insteadof the restoration filter 12 according to the first embodiment, andtherefore, the explanation is omitted herein.

Function Effect of Second Embodiment

In the digital camera 2 according to the second embodiment, thepoint-image restoration process is performed using the restorationfilter 62 generated based on the minimum value of the amplificationfactors of the restoration filters 64R, 64G, 64B for the respectivecolors at each spatial frequency and the function d_(K) (χ) that givesthe amplification factor of each MTF value of the restoration RGB imagedata with respect to the amplification factor x of the restorationfilter in the luminance system. That is, in the second embodiment, thepoint-image restoration process is performed using the restorationfilter 62 designed on the basis of a color having a low MTF value foreach spatial frequency, and therefore, each MTF value of the restorationRGB image data is suppressed to 1.0 or less. Further, since the functiond_(K) (χ) is used in the generation of the restoration filter 62, it ispossible to consider the difference in frequency amplification factoramong the respective planes of RGB in the luminance correction system.

Moreover, as shown in FIG. 15, since the point-image restoration processis performed using the restoration filter 62 designed on the basis of acolor having a low MTF value for each spatial frequency, the applicationof an excessive frequency emphasis in the low SN ratio region for the Bsignal is prevented unlike the case of using the restoration filter 12according to the first embodiment. As a result, it is possible tosuppress the amplification of the noise for the B signal. That is, it ispossible to prevent the amplification of the noise for another color (inthe embodiment, the B color) due to the frequency characteristic of arestoration filter (for example, the restoration filter 12) designed onthe basis of a color having a high MTF in a particular spatialfrequency.

Further, in the above Formula (5), the frequency characteristics h_(R),h_(G), h_(B) of the restoration filters 64R, 64G, 64B for the respectivecolors are determined based on the signal power S_(K) and noise powerN_(K) individualized for each color of RGB, and based on this result,the frequency characteristic f of the restoration filter 62 isdetermined by the above Formula (6). Thereby, it is possible toindividually set the assumed signal power and noise power for eachcolor, and it is possible to prevent the overcorrection that is causedby the difference in the signal and noise powers assumed for each color.

Third Embodiment

Next, a restoration filter generation device according to a thirdembodiment of the present invention is explained. As shown in FIG. 16,when the point-image restoration process is performed using the aboverestoration filter 62 according to the second embodiment, theapplication of an excessive frequency emphasis in the low SN ratioregion for the B signal is prevented, but on the contrary, frequencyemphasis is not applied in the frequency region for the R and G signalsthat corresponds to the low SN ratio region. Therefore, there is a fearthat the R and G signals attenuate. Further, as described above, whenthe point-image restoration process is performed using the aboverestoration filter 12 according to the first embodiment, there is a fearthat an excessive frequency emphasis is applied in the low SN ratioregion for the B signal and the noise for the B signal is amplified.

Hence, the restoration filter generation device according to the thirdembodiment generates a restoration filter 66 meeting an intermediatesolution that makes it possible to suppress both of the attenuation forthe R and G signals and the amplification of the noise for the B signal.

<Configuration of Restoration Filter Generation Device According toThird Embodiment>

As shown in FIG. 17, a restoration filter generation device 68 accordingto the third embodiment basically has the same configuration as thesecond embodiment, except that a device CPU 69 is included, and that arestoration filter generation program 70 (corresponding to a programaccording to the present invention) is stored in the storage 43.Therefore, for constituents having the same functions and configurationsas the above second embodiment, the same reference characters and thesame reference numerals are assigned, and the explanation is omitted.

When the restoration filter generation operation is performed in theoperation unit 41, the device CPU 69 reads the restoration filtergeneration program 70 from the storage 43, to execute it. Thereby, thedevice CPU 69 functions as a restoration filter generation unit(restoration filter generation device) 71, in addition to theabove-described MTF acquisition unit 50 and amplification factoracquisition unit 59.

<Restoration Filter Generation Process>

The restoration filter generation unit 71 generates a restoration filter66 in the luminance system that suppresses the overcorrection of the MTFvalue of the restoration RGB image data, that is, that suppresses theMTF value to 1.0 or less, based on the lens MTFs 47R, 47G, 47B and thefunction d_(K) (χ). On this occasion, the restoration filter generationunit 71 directly generates a restoration filter 66 meeting anintermediate solution that makes it possible to suppress both of theattenuation for the R and G signals and the amplification of the noisefor the B signal. In the following, the generation process of therestoration filter 66 is specifically explained.

The restoration filter generation unit 71 acquires the lens MTFs 47R,47G, 47B from the MTF acquisition unit 50, and acquires the functiond_(K) (χ) from the amplification factor acquisition unit 59.

Further, the restoration filter generation unit 71 determines thefrequency characteristic (MTF_(Y) (ω_(x), ω_(y))) of the lens MTF in theluminance system, based on already-known frequency characteristics(MTF_(K) (ω_(x), ω_(y)), Kε{R, G, B}) of the lens MTFs 47R, 47G, 47B.For example, MTF_(Y) (ω_(x), ω_(y)) is determined from the average orroot-mean-square of MTF_(R) (ω_(x), ω_(y)), MTF_(G) (ω_(x), ω_(y)) andMTF_(B) (ω_(x), ω_(y)).

Moreover, the restoration filter generation unit 71 determines thesignal power (S_(K) (ω_(x), ω_(y))) and noise power (N_(K) (ω_(x),ω_(y))) in the luminance system, based on already-known signal powers(S_(K) (ω_(x), ω_(y)), Kε{R, G, B}) and noise powers (N_(K) (ω_(x),ω_(y)), Kε{R, G, B}) for the respective colors, respectively. Similarlyto the above-described MTF_(Y) (ω_(x), ω_(y)), they are determined fromthe average or root-mean-square of S_(K) and N_(K) (Kε{R, G, B}) for therespective colors, respectively.

Subsequently, the restoration filter generation unit 71 starts thecomputation process for the generation of the restoration filter 66.When the filter coefficient of the restoration filter 66 with a tapnumber of N×N is represented by xεR^(N×N) and the frequencycharacteristic is represented by f (ω_(x), ω_(y)|x), the f (ω_(x),ω_(y)|x) is expressed as the following Formula (7). Here, N is an oddnumber, and M=(N−1)/2 is held. Further, each of u and v is a variablethat meets −M≦u, v≦M.

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 7} \rbrack & \; \\{{f( {\omega_{x},{\omega_{y}x}} )} = {\sum\limits_{u = {- M}}^{M}\; {\sum\limits_{v = {- M}}^{M}{e^{- {I{({{\omega_{x}u} + {\omega_{y}v}})}}}\lbrack x\rbrack}_{{u + M + 1},{v + M + 1}}}}} & (7)\end{matrix}$

As shown in the following Formula (8), the restoration filter generationunit 71 determines the filter coefficient x that minimizes a functionalJ [x] defined by the following Formula (9), under the conditionfollowing “s.t.” in Formula (8).

Here, “J [x]” in Formula (8) is the minimization criterion of the Wienerfilter to be used in the generation of the restoration filter. That is,the restoration filter generation unit 71 calculates the filtercoefficient x of such a Wiener filter that the MTF value of therestoration RGB image data for each color meets 1.0 or less, based onthe frequency characteristics (MTF_(K) (ω_(x), ω_(y)), K E {R, G, B}) ofthe lens MTFs 47R, 47G, 47B and the function d_(K) (χ). Thereby, in therestoration filter generation unit 71, the restoration filter 66 isgenerated. Similarly to the first and second embodiments, therestoration filter 66 is stored in the memory 14 of the digital camera2.

[Formula 8]

minimize J[x]s.t.∥d _(K)(f(ω_(x),ω_(y)|x))MTF_(K)(ω_(x),ω_(y))∥≦1,Kε{R,G,B}  (8)

[Formula 9]

J[x]=∫∫(∥1−f(ω_(x),Ω_(y)|)MTF_(J)(ω_(x),ω_(y))∥² S_(J)(ω_(x),ω_(y))+∥f(ω_(x),ω_(y) |x)∥² N _(J)(ω_(x),ω_(y)))dω _(x) dω_(y)  (9)

<Action of Restoration Filter Generation Device According to ThirdEmbodiment>

Next, the generation process of the restoration filter 66 by therestoration filter generation device 68 having the above configurationis explained using FIG. 18. When the restoration filter generationoperation is performed in the operation unit 41, the device CPU 69 readsthe restoration filter generation program 70 in the storage 43 toexecute it, and thereby, functions as the MTF acquisition unit 50, theamplification factor acquisition unit 59 and the restoration filtergeneration unit 71. Here, the process of inputting the lens MTFs 47R,47G, 47B and the function d_(K) (χ) respectively acquired by the MTFacquisition unit 50 and the amplification factor acquisition unit 59 tothe restoration filter generation unit 71 is basically the same as theabove second embodiment, and therefore, the explanation is omittedherein (step S20).

First, the restoration filter generation unit 71 determines thefrequency characteristic MTF of the lens MTF in the luminance system,the signal power S_(Y) and the noise power N_(Y), based on the frequencycharacteristics MTF_(K) of the lens MTFs 47R, 47G, 47B and the signalpowers S_(K) and noise powers N_(K) for the respective colors of RGB,respectively.

Subsequently, the restoration filter generation unit 71, as shown in theabove-described Formulas (8) and (9), calculates the filter coefficientx (the filter coefficient x of the Wiener filter) that minimizes thefunctional J [x] under the condition that the MTF value of therestoration RGB image data for each color meets 1.0 or less (step S25).Thereby, the restoration filter 66 is generated (step S26).

The restoration filter 66 generated by the restoration filter generationunit 71 is stored in the storage 43. Thereafter, similarly to the firstembodiment, the restoration filter 66 is stored in the memory 14 (stepS5).

<Action of Digital Camera According to Third Embodiment>

The imaging process in the digital camera 2 according to the thirdembodiment also is basically the same as the flow of the imaging processaccording to the first embodiment shown in FIG. 8, except that thepoint-image restoration process is performed using the restorationfilter 66 instead of the restoration filter 12 according to the firstembodiment, and therefore, the explanation is omitted herein.

Function Effect of Third Embodiment

As shown in FIG. 19, in the digital camera 2 according to the thirdembodiment, the point-image restoration process is performed using therestoration filter 66 generated by computing the filter coefficient xthat minimizes the functional J [x] under the condition that the MTFvalue of the restoration RGB image data for each color meets 1.0 orless. Thereby, each MTF value of the restoration RGB image data issuppressed to 1.0 or less.

Further, by computing the filter coefficient x that minimizes thefunctional J [x] under the above condition, it is possible to directlygenerate the restoration filter 66 optimized such that both of theattenuation for the R and G signals and the amplification of the noisefor the B signal are suppressed. In the second embodiment, in the casewhere the SN ratio for a particular color of RGB is low and the MTFvalue corresponding to the particular color is low, there is a fear thatthe restoration filter has a attenuation feature for suppressing thenoise emphasis and this brings the occurrence of a phenomenon in whichthe frequencies for all colors are brought to attenuation features.However, the third embodiment can prevent this. That is, in the thirdembodiment, it is possible to averagely perform the best frequencyrestoration while tolerating the noise emphasis for the particular colorin some degree, and to ensure that an excessive emphasis of the MTFvalue does not occur for any color.

Moreover, it is possible to prevent the overcorrection of the MTF valuethat is caused by the impossibility to achieve an ideal frequencycharacteristic due to the restriction of the tap number of therestoration filter.

Forth Embodiment

Next, a restoration filter generation device according to a fourthembodiment of the present invention is explained. In the aboverespective embodiments, each of the restoration filters 12, 62, 66 isgenerated based on the lens MTFs 47R, 47G, 47B. In contrast, in therestoration filter generation device according to the fourth embodiment,the generation of a restoration filter 73 (see FIG. 20 and FIG. 21) isperformed based on the lens MTF 47G for a particular color (in theembodiment, the G color).

<Configuration of Restoration Filter Generation Device According toFourth Embodiment>

As shown in FIG. 20, a restoration filter generation device 74 accordingto the fourth embodiment basically has the same configuration as thefirst embodiment, except that a device CPU 75 is included, and that arestoration filter generation program 76 (corresponding to a programaccording to the present invention) is stored in the storage 43.Therefore, for constituents having the same functions and configurationsas the above first embodiment, the same reference characters and thesame numerals are assigned, and the explanation is omitted.

When the restoration filter generation operation is performed in theoperation unit 41, the device CPU 75 reads the restoration filtergeneration program 76 from the storage 43, to execute it. Thereby, thedevice CPU 75 functions as an MTF acquisition unit (MTF acquisitiondevice) 77 and a restoration filter generation unit (restoration filtergeneration device) 78.

As shown in the (A) portion and (B) portion in FIG. 21, the MTFacquisition unit 77 selects the lens MTF 47G for the G color, from thelens MTFs 47R, 47G, 47B for the respective colors of RGB stored in thestorage 43, and outputs it to the restoration filter generation unit 78.Here, in the fourth embodiment, only the lens MTF 47G may be acquiredfrom the MTF measurement device 49, and the lens MTF 47G may be storedin the storage 43.

As shown in the (C) portion of FIG. 21, the restoration filtergeneration unit 78 generates the restoration filter 73 that can suppressthe overcorrection of the MTF value of the restoration RGB image datafor each color, based on the lens MTF 47G. Here, generally, lenses aredesigned with the emphasis on G, which has, of RGB, the greatestinfluence on the visual characteristic. Therefore, at least in afrequency region that is equal to or less than the half of the Nyquistfrequency of the color imaging element 23 (that is 0.25 Fs or less), thelens MTF 47G for the G plane is the highest of the lens MTFs 47R, 47G,47B. Therefore, when the restoration filter is designed such that thelens MTF 47G for the G plane is corrected, the overcorrection of the MTFvalue of the restoration RGB image data is prevented, at least in aregion that is equal to or less than 0.25 Fs (a particular spatialfrequency according to the present invention).

The restoration filter generation unit 78 computes the frequencycharacteristic of a Wiener filter corresponding to the lens MTF 47G,based on the frequency characteristic (MTF_(G) (ω_(x), ω_(y))) of thelens MTF 47G. Subsequently, based on the frequency characteristic of theWiener filter, the restoration filter generation unit 78 determines thefilter coefficient of the restoration filter 73 that achieves thefrequency characteristic in the luminance system. Here, as described inthe first embodiment, as the method for designing, from the frequencycharacteristic, the restoration filter (filter coefficient) thatachieves this, various known methods can be used. Thereby, in therestoration filter generation unit 78, the restoration filter 73 isgenerated. Similarly to the above respective embodiments, therestoration filter 73 is stored in the memory 14 of the digital camera2.

<Action of Restoration Filter Generation Device According to FourthEmbodiment>

Next, the generation process of the restoration filter 73 by therestoration filter generation device 74 having the above configurationis explained using FIG. 22. When the restoration filter generationoperation is performed in the operation unit 41, the device CPU 75 readsthe restoration filter generation program 76 in the storage 43 toexecute it, and thereby, functions as the MTF acquisition unit 77 andthe restoration filter generation unit 78. Here, on this occasion, atleast the lens MTF 47G is previously stored in the storage 43.

The MTF acquisition unit 77 acquires the lens MTF 47G from the storage43, and outputs the lens MTF 47G to the restoration filter generationunit 78 (step S28, MTF acquisition step).

The restoration filter generation unit 78 computes the frequencycharacteristic of the Wiener filter corresponding to the lens MTF 47G,based on the frequency characteristic of the lens MTF 47G (step S29),and thereafter, determines the filter coefficient of the restorationfilter 73 that achieves the frequency characteristic in the luminancesystem. Thereby, in the restoration filter generation unit 78, therestoration filter 73 is generated (step S30, restoration filtergeneration step).

The restoration filter 73 generated in the restoration filter generationunit 71 is stored in the storage 43. Thereafter, similarly to the firstembodiment, the restoration filter 73 is stored in the memory 14 (stepS5).

<Action of Digital Camera According to Fourth Embodiment>

The imaging process in the digital camera 2 according to the fourthembodiment also is basically the same as the flow of the imaging processin the first embodiment shown in FIG. 8, except that the point-imagerestoration process is performed using the restoration filter 73 insteadof the restoration filter 12 according to the first embodiment, andtherefore, the explanation is omitted herein.

Function Effect of Fourth Embodiment

According to the fourth embodiment, in the restoration filter generationdevice 74, the restoration filter 73 in the luminance system isgenerated based on the lens MTF 47G for the G color of the RGB, and inthe digital camera 2, the point-image restoration process is performedusing the restoration filter 73. At least in the frequency region thatis equal to or less than the half of the Nyquist frequency (that is 0.25Fs or less), the lens MTF 47G has the highest MTF value, of the lensMTFs 47R, 47G, 47B. Therefore, when the point-image restoration processis performed using the restoration filter 73 generated based on the lensMTF 47G, the overcorrection of the MTF value of the RGB signal componentof the luminance system image data Y after the point-image restorationprocess is prevented at least in the frequency region of 0.25 Fs orless. That is, in the optical system 17, when the modulation transferfunction MTF for the G color is higher than the modulation transferfunctions MTF for the other colors at least in the frequency region of0.25 Fs or less, the overcorrection of the MTF value of the image datafor each color of the multiple colors corresponding to the luminancesystem image data after the restoration process is prevented at least inthe frequency region of 0.25 Fs or less. Therefore, even when therestoration filter 73 is generated using the lens MTF 47G instead of thelens MTF with respect to the luminance, a certain degree of effect ofsuppressing the MTF value of the restoration RGB image data for eachcolor corresponding to the luminance system image data Y after thepoint-image restoration process to 1.0 or less is expected at least inthe frequency region of 0.25 Fs or less. Particularly, the greater thedifference of the value of the lens MTF 47G for the G color from thevalues of the lens MTF 47R and lens MTF 47B for the R color and B coloris, the more the effect is obtained.

Further, in the fourth embodiment, since the restoration filter 73 isgenerated based on the lens MTF 47G for the G color of RGB, it ispossible to simplify the computation process required for the generationof the restoration filter (to simplify the design procedure of therestoration filter), compared to the above respective embodiments.Further, unlike the above respective embodiments, it is possible toperform the generation of the restoration filter 73 without acquiringthe lens MTFs 47R, 47G, 47B for all colors of RGB.

Here, generally, lenses are designed such that the lens MTF 47Gcorresponding to the G color, which has the greatest influence on thevisual characteristic, is highest at least in the frequency region of0.25 Fs or less, and therefore, in the above fourth embodiment, theparticular spatial frequency in the present invention is prescribed as0.25 Fs or less. Therefore, in the case where the frequency region inwhich the lens MTF 47G corresponding to the G color is highest isincreased or decreased from 0.25 Fs or less by the design of the lens,the particular spatial frequency in the present invention, in responseto this, may be also increased or decreased.

Further, in the above fourth embodiment, the restoration filter 73 isgenerated based on the lens MTF 47G corresponding to the G color.However, in the case where the color filters of the color imagingelement 23 include a color other than RGB, the restoration filter may begenerated based on the lens MTF corresponding to the color that mostcontributes to the luminance, for example.

OTHER APPLICATION EXAMPLES <Application Example to EDoF System>

The point-image restoration processes according to the above respectiveembodiments are image processes (restoration processes) of restoring theoriginal subject image by recovering and correcting the imagedegradation (point-image blurring, point spread) depending on aparticular image-taking condition (for example, the diaphragm value, theF-value, the focal distance, the lens type and the like), but thepoint-image restoration process to which the present invention can beapplied is not limited to this. For example, the point-image restorationprocess according to the present invention can be applied also to therestoration process to the image data imaged by an imaging device thatincludes an optical system (an image-taking lens and the like) having anextended depth of field (focus) (EDoF). By performing the restorationprocess on the image data of a blurred image taken and acquired in astate in which the depth of field (depth of focus) is extended by theEDoF optical system, it is possible to restore and generate the imagedata that is in an in-focus state over a wide range and that gives ahigh resolution feeling. In this case, there is performed a restorationprocess using a restoration filter that is based on a point spreadfunction (a PSF, an OTF (Optical Transfer Function), an MTF, a PTF(Phase Transfer Function), or the like) for the EDoF optical system andthat has a filter coefficient set so as to allow for a suitable imagerestoration in the range of the extended depth of field (depth offocus).

In the following, an example of a system (EDoF system) relevant to therestoration of the image data taken and acquired through the EDoFoptical system is explained. Here, in the example shown below, anexample in which the restoration process is performed on the luminancesystem image data Y obtained from the image data (RGB image data) afterthe demosaic process is explained. However, the timing to perform therestoration process is not particularly limited, and the restorationprocess may be performed, for example, to the “image data before thedemosaic process (mosaic image data)” or the “image data after thedemosaic process and before the luminance signal conversion process(demosaic image data).”

FIG. 23 is a block diagram showing a form of an imaging module 101including an EDoF optical system. An imaging module (imaging device) 101in the example includes an EDoF optical system (optical system) 110, aCCD type color imaging element 112 (a CMOS type may be adopted, imagingdevice), an AD conversion unit 114, and a restoration processing unit120 that functions as a restoration processing device according to thepresent invention.

FIG. 24 is a diagram showing an example of the EDoF optical system 110.The EDoF optical system 110 includes fixed single-focus image-takinglenses 110A, and an optical filter 111 arranged at a pupil position. Inorder to obtain an extended depth of field (depth of focus) (EDoF), theoptical filter 111, which modulates the phase, organizes (puts into anEDoF state) the EDoF optical system 110 (the image-taking lens 110A).Thus, the image-taking lens 110A and the optical filter 111 constitute alens unit that modulates the phase and extends the depth of field.

Here, the EDoF optical system 110 includes other constituent elements asnecessary, and for example, a diaphragm (not shown) is disposed at thevicinity of the optical filter 111. Further, the optical filter 111 maybe a single filter, or may be a combination of multiple filters.Further, the optical filter 111 is just an example of an optical phasemodulation device, and the EDoF state of the EDoF optical system 110(the image-taking lens 110A) may be achieved by other devices. Forexample, instead of providing the optical filter 111, the EDoF state ofthe EDoF optical system 110 may be achieved by an image-taking lens 110Athat is lens-designed so as to have a function equivalent to the opticalfilter 111 in the example. That is, the EDoF state of the EDoF opticalsystem 110 can be achieved by a variety of devices for changing thewave-front of the image formation on the light receiving surface of theimaging element 112. For example, an “optical element whose thickness ischangeable,” an “optical element whose refractive index is changeable (arefractive-index distribution type wave-front modulation lens, or thelike),” an “optical element whose thickness or refractive index ischangeable by the coding on the lens surface, or the like (a wave-frontmodulation hybrid lens, an optical element to be formed on the lenssurface as a phase plane, or the like),” and a “liquid crystal elementcapable of modulating the phase distribution of light (a liquid-crystalspace-phase modulation element or the like)” can be employed as a devicefor putting the EDoF optical system 110 into the EDoF state. Thus, thepresent invention can be applied to not only the case where a regularlydispersed image can be formed by a light wave-front modulation element(the optical filter 111 (phase plate)), but also the case where the samedispersed image as the case of using the light wave-front modulationelement can be formed by the image-taking lens 110A itself without usingthe light wave-front modulation element.

The EDoF optical system 110 according to the embodiment allows fordownsizing, and can be suitably mounted in a camera-equipped mobilephone or a portable information terminal, because a focus adjustmentmechanism to mechanically perform a focus adjustment can be omitted.

Returning to FIG. 23, an optical image after passing through the EDoFoptical system 110 in an EDoF state is formed on the imaging surface ofthe color imaging element 112, and here, is converted into an electricsignal.

The color imaging element 112 basically has the same configuration asthe color imaging element 23 according to the above respectiveembodiments. The CCD type color imaging element 112 converts the subjectlight formed on the imaging surface by the EDoF optical system 110, intosignal charges of a quantity corresponding to the incident lightquantity, and outputs an analog RGB image signal.

The AD conversion unit 114 converts the analog RGB image signal outputfrom the color imaging element 112, into digital mosaic image data foreach color of RGB. The mosaic image data for each color is input to therestoration processing unit 120.

The restoration processing unit 120 includes, for example, a black leveladjustment unit 122, a white balance gain unit 123, a gamma processingunit 124, a demosaic processing unit 125, an RGB/YCrCb conversion unit(hereinafter, abbreviated to a conversion unit, image data generationdevice) 126, and a Y signal restoration processing unit (restorationprocessing device) 127.

The black level adjustment unit 122 performs a black level adjustment onthe mosaic image data for each color output from the AD conversion unit114. As the black level adjustment, a known method can be employed. Forexample, in the case of focusing attention on an effective photoelectricconversion element, the black level adjustment is performed bydetermining the average of dark current amount acquisition signalsrespectively corresponding to multiple OB photoelectric conversionelements contained in a photoelectric conversion element line containingthe effective photoelectric conversion element and subtracting theaverage from a dark current amount acquisition signal corresponding tothe effective photoelectric conversion element.

The white balance gain unit 123 performs a gain adjustment correspondingto the white balance gain of the mosaic image data for each color afterthe black level data adjustment.

The gamma processing unit 124 performs a gamma correction for thegradation correction such as half tone such that the mosaic image datafor each color after the white balance adjustment has an intended gammacharacteristic.

The demosaic processing unit 125 performs a demosaic process on themosaic image data for each color after the gamma correction, and outputsthe RGB image data composed of the color data of the three planes ofRGB.

The conversion unit 126, which is basically the same as the conversionunit 37 according to the above respective embodiments, performs the YCconversion process on the RGB image data output from the demosaicprocessing unit 125, and generates and outputs the luminance systemimage data Y and the color-difference system image data Cb, Cr.

The Y signal restoration processing unit 127 performs the restorationprocess on the luminance system image data Y from the conversion unit126, based on a previously stored restoration filter. The restorationfilter, for example, has a deconvolution kernel having a kernel size of7×7 (corresponding to a tap number of M=7, N=7), and a computationcoefficient (restoration gain data, corresponding to the filtercoefficient) corresponding to the deconvolution kernel, and is used in adeconvolution process (deconvolution computation process) by the phasemodulation amount of the optical filter 111. Here, the restorationfilter corresponds to the optical filter 111, and is stored in a memorynot illustrated. Further, the kernel size of the deconvolution kernel isnot limited to the size of 7×7.

Next, the restoration process by the restoration processing unit 120 isexplained using the flowchart shown in FIG. 25.

The mosaic image data for each color is input from the AD conversionunit 114 to one input of the black level adjustment unit 122, and theblack level data is input to another input. The black level adjustmentunit 122 subtracts the black level data from the mosaic image data foreach color, and outputs the mosaic image data for each color after thesubtraction process, to the white balance gain unit 123 (step S31).Thereby, the mosaic image data for each color no longer includes theblack level component.

To the mosaic image data for each color after the black leveladjustment, the processes by the white balance gain unit 123 and thegamma processing unit 124 are sequentially performed (steps S32 andS33).

The mosaic image data for each color after the gamma correction isconverted in the conversion unit 126 into the luminance system imagedata Y and the color-difference system image data Cb, Cr (step S34),after the demosaic process by the demosaic processing unit 125.

The Y signal restoration processing unit 127 performs, on the luminancesystem image data Y, the restoration process of applying thedeconvolution process by the phase modulation amount of the opticalfilter 111 of the EDoF optical system 110 (step S35). That is, the Ysignal restoration processing unit 127 performs the deconvolutionprocess (deconvolution computation process) between a luminance signalcorresponding to a predetermined unit pixel group around an arbitraryprocess-target pixel (herein, a luminance signal for 7×7 pixels) and arestoration filter (a deconvolution kernel of 7×7 and the computationcoefficient) that is previously stored in the memory or the like. The Ysignal restoration processing unit 127 performs the restoration processof removing the image blurs of the whole image by repeating thedeconvolution process for each of the predetermined unit pixel groups soas to cover the whole region of the imaging surface. The restorationfilter is defined depending on the center position of the pixel groupfor which the deconvolution process is performed. That is, a commonrestoration filter is applied to adjacent pixel groups. For furthersimplifying the restoration process, it is preferable to apply a commonrestoration filter to all pixel groups.

The point-image (optical image) of the luminance signal after passingthrough the EDoF optical system 110 is formed on the color imagingelement 112, as a large point-image (a blurred image) (see the (A)portion of FIG. 5), and but is restored to a small point-image (an imagegiving a high resolution feeling) by the deconvolution process in the Ysignal restoration processing unit 127 (see the (B) portion of FIG. 5).

As described above, the restoration process is applied to the luminancesystem image data Y after the demosaic process, and therefore, it isunnecessary to have a parameter of the restoration process for each ofRGB, allowing for the speeding up of the restoration process. Further,the deconvolution process is performed by arranging the individualluminance signals of adjacent pixels as the predetermined unit andapplying a common restoration filter to the unit, instead of performingthe deconvolution process by arranging, as one unit, each of theindividual R, G and B image signals corresponding to R, G and B pixelsthat are positioned at intervals. Therefore, the accuracy of therestoration process is enhanced. Here, as for the color-differencesystem image data Cb, Cr, even when the resolution feeling is notincreased by the restoration process, the image quality is allowedbecause of the characteristic of the visual sensation of human eyes.Further, in the case where the image is recorded in a compression formatsuch as JPEG (Joint Photographic Experts Group), the color-differencesystem image data Cb, Cr are compressed at a higher compressibility thanthat of the luminance system image data Y, and therefore, it is lessnecessary to increase the resolution by the restoration process. Thus,it is possible to attain both the enhancement of the restorationaccuracy and the simplification and speeding up of the process.

The restoration processes according to the respective embodiments of thepresent invention can be applied also to the restoration process of theEDoF system as explained above. In this case, based on the MTF for theEDoF optical system, the restoration filter generation devices accordingto the above respective embodiments generate a restoration filter havinga filter coefficient that is set so as to allow for a suitable imagerestoration in the range of the extended depth of field (depth offocus).

<Application Example to Smart Phone>

In the above respective embodiments, the digital camera 2 has beenexplained as an example of the image processing device and the imagingdevice according to the present invention. However, the presentinvention can be applied also to a mobile phone, a smart phone, a PDA(Personal Digital Assistants), a tablet terminal and a portable gamemachine that have an image-taking function, for example. In thefollowing, as an example, a smart phone is explained in detail withreference to the drawings.

FIG. 26 shows an external view of a smart phone 500. The smart phone 500has a case 501 in a flat plate shape. It includes, on one surface of thecase 501, a display/input unit 502, a speaker 503, a microphone 504, anoperation unit 505 and a camera unit 506. Here, the configuration of thecase 501 is not limited to this, and for example, a configuration inwhich the display unit and the input unit are separated, or aconfiguration having a folding structure or a sliding mechanism can bealso employed. Further, the camera unit 506 is provided also on theother surface of the case 501.

The display/input unit 502 displays an image (a still image and a movingimage), character information and the like. Further, the display/inputunit 502 has a so-called touch panel structure in which a user operationto the displayed information is detected. The display/input unit 502 iscomposed of a display panel 510 and an operation panel 512.

The display panel 510 is an LCD (Liquid Crystal Display), an OELD(Organic Electro-Luminescence Display) or the like that is used as adisplay device. The operation panel 512, which has optical transparency,is placed on the display surface of the display panel 510. The operationpanel 512 is a device to detect a single or multiple coordinates thatare operated by a finger of a user or a stylus. When this device isoperated by a finger of a user or a stylus, a detection signal generateddue to the operation is output to a CPU of the smart phone 500. The CPUdetects the operation position (coordinate) on the display panel 510,based on the received detection signal. As the position detection schemeto be employed in such an operation panel 512, there are a matrix switchscheme, a resistive film scheme, a surface acoustic wave scheme, aninfrared ray scheme, an electromagnetic induction scheme, anelectrostatic capacity scheme and the like.

As shown in FIG. 27, the smart phone 500 includes a wirelesscommunication unit 515, a telephone call unit 516, a storage unit 517,an external input/output unit 518, a GPS (Global Positioning System)receiving unit 519, a motion sensor unit 520 and a power unit 521, inaddition to the display/input unit 502, the speaker 503, the microphone504, the operation unit 505, the camera unit 506, the CPU 507 and adisplay processing unit 508.

The operation unit 505 is a hardware key in which a push-button switch,a cross key or the like is used for example, and receives an instructionfrom a user. The operation unit 505 is mounted on a lower part of thedisplay unit of the case 501 or a side surface of the case 501, forexample.

The camera unit 506 basically has the same configuration as the digitalcamera 2 according to the above respective embodiments. In the memory 14of the camera unit 506, any one of the restoration filters 12, 62, 66,73 according to the above respective embodiments is stored.

The display processing unit 508 displays an image or characterinformation on the display/input unit 502, in accordance with aninstruction of the CPU 507.

The wireless communication unit 515 performs the wireless communicationwith a base station device contained in a mobile communication network,in accordance with an instruction of the CPU 507. Using the wirelesscommunication, the sending and receiving of a variety of file data suchas audio data and image data, e-mail data or the like, and the receivingof Web data, streaming data or the like are performed.

The telephone call unit 516 includes the speaker 503 and the microphone504. The telephone call unit 516 converts a user voice input through themicrophone 504 into audio data, to output it to the CPU 507, and decodesthe audio data received by the wireless communication unit 515 or thelike, to output it from the speaker 503.

The storage unit 517 stores the control program and control data of theCPU 507, application software, address data associated with the name,telephone number and others of a communication partner, sent or receivede-mail data, and the like, and temporarily stores streaming data and thelike. Further, the storage unit 517 is composed of an internal storageunit 517 a built in the smart phone and an external storage unit 517 bhaving a detachable external memory slot. Here, as the internal storageunit 517 a and the external storage unit 517 b, various known storagemedia such as a flash memory type and a hard disc type are used.

The external input/output unit 518 serves as an interface with allexternal apparatuses that are linked with the smart phone 500, and isdirectly or indirectly connected with other external apparatuses,through communication or the like.

The GPS receiving unit 519 receives GPS signals sent from GPS satellitesST1 to STn, executes a positioning computation process based on themultiple GPS signals received, and detects the position of the smartphone 500 as the latitude, the longitude and the altitude. The detectionresult is output to the CPU 507.

The motion sensor unit 520 includes, for example, a three-axisacceleration sensor and the like, and detects the physical motion of thesmart phone 500. Thereby, the moving direction and acceleration of thesmart phone 500 are detected. The detection result is output to the CPU507. Further, the power unit 521 supplies the electric power stored in abattery not illustrated, to each unit of the smart phone 500.

The CPU 507 operates in accordance with the control program and controldata read from the storage unit 517, and integrally controls each unitof the smart phone 500. Further, the CPU 507 executes the displaycontrol to the display panel 510, the operation detection control fordetecting a user operation through the operation unit 505 or theoperation panel 512 and the like.

By the execution of the display control, the CPU 507 displays, softwarekeys such as an icon for activating application software and a scrollbar, or displays a window for creating an e-mail message, or the like onthe display panel 510. Here, the scroll bar means a software key forreceiving an instruction to move the display portion of an image that istoo large to fit inside the display region of the display panel 510, orthe like.

Further, by the execution of the operation detection control, the CPU507 detects a user operation through the operation unit 505, receives anoperation to the above icon and an input of a character string to aninput box of the above window, through the operation panel 512, orreceives a scroll request of a display image through the scroll bar.

Moreover, by the execution of the operation detection control, the CPU507 has a touch panel control function to decide whether the operationposition to the operation panel 512 is a superimposition portion(display region) that overlaps with the display panel 510, or an outeredge portion (non-display region) that is other than this and that doesnot overlap with the display panel 510, and to control the sensitiveregion of the operation panel 512 and the display positions of thesoftware keys.

Further, the CPU 507 can detect a gesture operation to the operationpanel 512, and can execute a previously set function in response to thedetected gesture operation. The gesture operation means not aconventional simple touch operation, but an operation to draw a trackfrom at least one of multiple positions by drawing the track by a fingeror the like, by designating multiple positions simultaneously, or bycombining them.

As described above, the camera unit 506 of the smart phone 500 havingthe above configuration basically has the same configuration as thedigital camera 2 according to the above respective embodiments, andtherefore, the same effects as the above respective embodiments areobtained.

{Addition}

In the above respective embodiments, the imaging devices having animage-taking function, as exemplified by the digital camera 2 and thesmart phone 500, have been explained as examples of the image processingdevice to perform the restoration process according to the presentinvention, but the image-taking function is not essential. For example,the present invention can be applied also to various image processingdevices to perform the restoration process on the image data acquiredthrough the internet or a recording medium, as exemplified by a tabletterminal, a smart phone and a PC.

The present invention can be provided, for example, as acomputer-readable program code for making an imaging device or anelectronic apparatus including an imaging device perform the aboveprocess, a non-transitory and computer-readable recording medium inwhich the program code is stored (for example, an optical disc (forexample, a CD (Compact Disc), a DVD (Digital Versatile Disc) and a BD(Blu-ray® Disc)), a magnetic disc (for example, a hard disc and amagneto-optical disc), and a USB (Universal Serial Bus) memory), and acomputer program product in which an executable code for the method isstored.

In the above respective embodiments, the digital camera 2, the smartphone 500 and the like corresponding to the image processing device andthe imaging device according to the present invention store therestoration filters that are separately generated in the restorationfilter generation devices according to the above respective embodiments.However, the digital camera 2 or the like, and the restoration filtergeneration device may be unified.

What is claimed is:
 1. A restoration filter generation device whichgenerates a restoration filter for performing a restoration process onluminance system image data, the restoration process being based on apoint-image distribution in an optical system, the luminance systemimage data being image data relevant to luminance and being generatedbased on image data for each color of multiple colors, the image datafor each color of the multiple colors being obtained by an imagingdevice including the optical system, the restoration filter generationdevice comprising: a modulation transfer function MTF acquisition devicewhich acquires a modulation transfer function MTF for the opticalsystem; and a restoration filter generation unit which generates therestoration filter based on the modulation transfer function MTFacquired by the MTF acquisition device, the restoration filtersuppressing an MTF value of image data for each color of the multiplecolors to 1.0 or less at least in a region of a particular spatialfrequency or less, the image data for each color of the multiple colorscorresponding to the luminance system image data after the restorationprocess, wherein: the image data for the multiple colors contains imagedata for each color of RGB; the MTF acquisition device acquires themodulation transfer function MTF for the G color; and the restorationfilter generation unit determines a frequency characteristic of a Wienerfilter based on the modulation transfer function MTF for the G coloracquired by the MTF acquisition device, and generates the restorationfilter that achieves the frequency characteristic in a luminance system.2. The restoration filter generation device according to claim 1,wherein the particular spatial frequency is equal to or less than a halfof a Nyquist frequency of an imaging element included in the imagingdevice.
 3. The restoration filter generation device according to claim1, wherein the MTF acquisition device acquires the modulation transferfunction MTF for the optical system including a lens unit that modulatesa phase and extends a depth of field.
 4. An image processing devicecomprising: an image data generation device which generates luminancesystem image data based on image data for each color of multiple colors,the luminance system image data being image data relevant to luminance,the image data for each color of the multiple colors being obtained byan imaging device including an optical system; a restoration filterstorage device which stores the restoration filter generated by therestoration filter generation device according to claim 1; and arestoration processing device which performs a restoration process onthe luminance system image data generated by the image data generationdevice, using the restoration filter stored in the restoration filterstorage device.
 5. An imaging system comprising: an imaging device whichoutputs image data for each color of multiple colors, the imaging deviceincluding an optical system; and the image processing device accordingto claim
 4. 6. A restoration filter generation method for generating arestoration filter for performing a restoration process on luminancesystem image data, the restoration process being based on a point-imagedistribution in an optical system, the luminance system image data beingimage data relevant to luminance and being generated based on image datafor each color of multiple colors, the image data for each color of themultiple colors being obtained by an imaging device including theoptical system, the restoration filter generation method comprising: amodulation transfer function MTF acquisition step of acquiring amodulation transfer function MTF for the optical system; and arestoration filter generation step of generating the restoration filterbased on the modulation transfer function MTF acquired in the MTFacquisition step, the restoration filter suppressing an MTF value ofimage data for each color of the multiple colors to 1.0 or less at leastin a region of a particular spatial frequency or less, the image datafor each color of the multiple colors corresponding to the luminancesystem image data after the restoration process, wherein: the image datafor the multiple colors contains image data for each color of RGB; theMTF acquisition step acquires the modulation transfer function MTF forthe G color; and the restoration filter generation step determines afrequency characteristic of a Wiener filter based on the modulationtransfer function MTF for the G color acquired by the MTF acquisitionstep, and generates the restoration filter that achieves the frequencycharacteristic in a luminance system.
 7. An image processing methodcomprising: an image data generation step of generating luminance systemimage data based on image data for each color of multiple colors, theluminance system image data being image data relevant to luminance, theimage data for each color of the multiple colors being obtained by animaging device including an optical system; and a restoration processingstep of performing a restoration process on the luminance system imagedata generated in the image data generation step, using the restorationfilter generated by the restoration filter generation method accordingto claim
 6. 8. A non-transitory computer-readable medium recording aprogram for generating a restoration filter for performing a restorationprocess on luminance system image data, the restoration process beingbased on a point-image distribution in an optical system, the luminancesystem image data being image data relevant to luminance and beinggenerated based on image data for each color of multiple colors, theimage data for each color of the multiple colors being obtained by animaging device including the optical system, the program causing acomputer to execute: a modulation transfer function MTF acquisition stepof acquiring a modulation transfer function MTF for the optical system;and a restoration filter generation step of generating the restorationfilter based on the modulation transfer function MTF acquired in the MTFacquisition step, the restoration filter suppressing an MTF value ofimage data for each color of the multiple colors to 1.0 or less at leastin a region of a particular spatial frequency or less, the image datafor each color of the multiple colors corresponding to the luminancesystem image data after the restoration process, wherein: the image datafor the multiple colors contains image data for each color of RGB; theMTF acquisition step acquires the modulation transfer function MTF forthe G color; and the restoration filter generation step determines afrequency characteristic of a Wiener filter based on the modulationtransfer function MTF for the G color acquired by the MTF acquisitionstep, and generates the restoration filter that achieves the frequencycharacteristic in a luminance system.